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ABSTRACT 


Designing  naval  ship  capabilities  for  shipbuilding  is  a  challenging  process 
requiring  comprehensive  technical  and  tactical  studies.  Technical  studies  involve 
ship  design  characteristics  such  as  engineering,  weapon,  and  support  systems. 
Tactical  studies  include  the  anticipated  area  of  operation,  expected  threat,  the 
capabilities  of  the  enemy,  and  potential  missions  to  accomplish.  Both  studies  are 
used  in  ship  design  to  determine  the  ship’s  required  combat  capabilities  before 
finalizing  the  hull  design.  This  research  uses  the  agent-based  modeling  tool  Map 
Aware  Non-Uniform  Automata  (MANA)  to  explore  the  best  combat  capabilities  for 
a  frigate  in  an  anti-air  warfare  (AAW)  environment.  Regression  and  partition  trees 
are  used  to  analyze  factors  that  influence  the  measures  of  the  friendly  frigate’s 
survivability  and  number  of  enemy  casualties.  This  study  also  investigates  the 
use  of  a  prospective  ship-based  unmanned  aerial  vehicle  in  AAW  operations.  We 
find  that  the  inclusion  of  a  Point  Defense  Missile  System  with  long  and  medium 
range  surface-to-air  missiles  has  the  most  positive  effects  on  ship  survivability. 
By  contrast,  we  find  the  inclusion  of  a  UAV  in  this  mission  has  little  effect. 
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EXECUTIVE  SUMMARY 


The  combat  capabilities  of  a  future  naval  ship  have  to  be  considered 
thoroughly  in  the  ship  design  process.  These  capabilities  should  be  determined 
before  the  ship’s  hull  design  is  complete  to  make  operational  effectiveness 
independent  from  physical  design  considerations.  The  use  of  simulation  with 
advanced  experimental  designs  provides  useful  insights  about  the  required 
combat  systems  in  expected  missions  the  ship  must  undertake. 

This  thesis  uses  the  Map  Aware  Non-Uniform  Automata  (MANA)  combat 
modeling  tool  to  identify  the  best  combination  of  weapons  and  radar 
characteristics  onboard  a  frigate  in  an  anti-air  warfare  (AAW)  environment.  The 
effectiveness  of  a  prospective  ship-based  unmanned  aerial  vehicle  (UAV)  in  an 
AAW  operation  is  evaluated,  as  well.  We  develop  an  AAW  scenario  in  MANA  to 
investigate  key  weapons,  sensor,  and  UAV  effectiveness  on  ship  survivability 
and  enemy  casualties.  The  MANA  scenario  representation  appears  in  the 
following  figure.  It  shows  a  lone  AAW  frigate  threatened  by  missile-carrying 
aircraft  and  land  launched  anti-ship  missiles  from  the  shore. 


Figurel .  Screenshot  of  the  AAW  Tactical  Scenario  Modeled  in  MANA 

xvii 


The  frigate’s  combat  defense  system  is  limited  to  four  weapon  systems 
due  to  space  restrictions.  A  gun  system  is  always  preferred  to  be  onboard  due  to 
its  versatility  in  a  variety  of  missions.  The  other  three  weapon  systems  consist  of 
two  types  of  surface-to-air  missile  systems  (SAM)  and  either  a  Close-in  Weapon 
System  Gatling  gun  (CIWS)  or  a  Point  Defense  Missile  System  (PDMS).  These 
three  weapon  systems  are  changed  in  the  experimental  design  to  determine  the 
best  mix  of  weapon  types  for  an  AAW  frigate  in  this  tactical  scenario.  For  the 
evaluation  of  the  sensor  capabilities  we  also  vary  the  range  of  the  radar  from 
40,000  meters  to  250,000  meters  in  the  experimental  design.  The  candidate 
weapons  and  their  specifications  are  as  shown  in  the  following  table. 


Table  1 .  Candidate  Weapon  Types  and  Specifications 


Weapon 

Minimum  Range 

/  P  (Hit) 

Maximum  Range 

/  P  (Hit) 

Target  Type 

CIWS 

1,000  m/  0.6 

6,000  m/  0.4 

Only  missiles 

PDMS 

1,000  m/  0.8 

9,000  m/  0.6 

Only  missiles 

Gun 

8,000  m/  0.3 

14,000  m/  0.2 

Both  aircraft  and 

missiles 

Short  Range  SAM 

5,000  m/  0.8 

30,000  m/  0.3 

Both  aircraft  and 

missiles 

Medium  Range 

SAM 

10,000  m/  0.8 

70,000  m/  0.1 

Both  aircraft  and 

missiles 

Long  Range  SAM 

10,000  m/  0.8 

200,000  m/ 0.1 

Both  aircraft  and 

missiles 

We  use  regression  analysis  and  partition  trees  to  analyze  the  results  of 
25,700  simulated  battles.  The  analysis  shows  that  the  most  important  design 


factors  for  frigate  AAW  operations  is  the  selection  of  CIWS  or  PDMS. 
Furthermore,  it  shows  that  the  PDMS  is  superior  to  CIWS  in  enhancing  ship 
survivability  and  inflicting  enemy  casualties.  This  study  also  demonstrates  that 
longer  range  SAMs,  the  combination  of  the  SAMs  onboard,  and  longer  radar 
range  have  significant  impacts  on  the  success  of  an  AAW  operation.  In  addition, 
this  research  provides  evidence  that  the  use  of  a  medium  range  UAV  in  an  AAW 
environment  does  not  substantially  contribute  to  mission  success.  Longer  range 
UAVs  with  greater  surveillance  capabilities  may  have  more  effect,  but  they  are 
not  explored  in  this  study. 
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I.  INTRODUCTION 


A.  OVERVIEW 

Designing  naval  ship  capabilities  for  shipbuilding  is  a  challenging  process 
requiring  comprehensive  technical  and  tactical  studies.  Technical  studies  involve 
ship  design  characteristics  such  as  engineering,  weapon,  and  support  systems. 
Tactical  studies  include  the  anticipated  area  of  operation,  expected  threat,  the 
capabilities  of  the  enemy,  and  potential  missions  to  accomplish. 

Designing  a  naval  ship  is  also  a  complicated  process  because  it  requires 
developing  a  system  of  systems  (Mizine,  Wintersteen,  &  Wynn,  2012).  This 
article  also  states  that  since  these  systems  interact  and  influence  each  other  the 
developmental  process  is  even  more  challenging.  If  we  consider  an  entire  fleet 
architecture  rather  than  an  individual  ship  design,  this  challenging  work  becomes 
further  complicated.  Moreover,  a  variety  of  expected  threats  need  to  be 
considered  during  the  ship  design  process  as  well.  For  example,  in  an  anti-air 
warfare  (AAW)  mission,  weapons  and  radar  systems  influence  each  other  and 
interact  with  the  combat  software  system  (Figure  1). 


Figure  1 .  A  Naval  Ship  in  an  AAW  Environment. 
Source:  Oneindia  News  (2015). 
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Successfully  defending  naval  ships  from  air  threats  depends  mainly  on 
force  capabilities  and  the  tactics  of  each  opposing  side.  Considering  the  air 
defense  capabilities  of  our  forces  against  an  enemy,  an  AAW  frigate  should  be 
able  to  detect  and  eliminate  enemy  aircraft  and  guided  missiles  while  deterring 
other  threats,  such  as  submarines,  small  craft,  and  other  surface  ships.  To 
achieve  this  tactical  objective,  the  ship’s  weapon  configuration  should  be 
determined  as  part  of  new  ship  construction  design. 

There  is  limited  space  onboard  a  frigate;  therefore,  selecting  the  most 
effective  combination  of  weapon  systems  to  be  included  in  the  ship’s  design  is  a 
matter  of  great  importance. 

B.  RESEARCH  QUESTIONS 

This  research  focuses  on  the  following  questions  related  to  the 
improvement  of  the  ship  design  process: 

•  Among  a  set  of  air  defense  weapon  systems  alternatives,  what  is 
the  most  effective  combination  for  a  future  AAW  frigate  design? 

•  How  effective  are  Point  Defense  Missile  Systems  (PDMS) 
compared  to  Close-in  Weapon  Systems  (CIWS)  with  different 
weapon  configurations? 

•  Does  employing  an  unmanned  aerial  vehicle  (UAV)  onboard  an 
AAW  frigate  have  significant  advantages  in  an  AAW  mission? 

•  What  is  the  probability  of  survivability  against  enemy  air  assets  with 
different  weapon  combinations? 

•  What  are  the  strengths  and  drawbacks  of  utilizing  Map  Aware  Non- 
Uniform  Automata  (MANA)  to  construct  realistic  scenarios  for 
evaluating  AAW  effectiveness  of  naval  ships? 

C.  SCOPE  OF  THE  THESIS  AND  METHODOLOGY 

This  thesis’  main  focus  is  evaluating  the  effectiveness  of  different 
combinations  of  weapon  types  and  radars  onboard  a  frigate  in  a  realistic  AAW 
environment  with  the  intent  to  inform  Turkish  AAW  frigate  design.  This  study  also 
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explores  how  a  UAV  like  the  MQ-9  (Figure  2)  can  contribute  to  mission  success 
by  providing  early  warning  to  surface  ships. 


Figure  2.  USS  Fort  Worth  (LCS-3)  Launches  First  UAV. 
Source:  America's  Navy  (2013). 


We  use  an  agent-based  simulation  modeling  platform  called  MANA  to 
model  AAW  scenarios,  allowing  us  to  assess  the  defensive  capabilities  of 
alternative  combat  system  configurations  on  a  future  AAW  frigate.  “Agent-based 
simulations  are  models  where  multiple  entities  sense  and  stochastically  respond 
to  conditions  in  their  local  environments,  mimicking  complex  large-scale  system 
behavior”  (Sanchez  &  Lucas,  2002). 

We  create  a  base  case  tactical  situation,  threat,  and  combatant 
configuration  in  MANA  for  comparing  alternatives.  In  the  base  case,  a  single 
AAW  frigate  is  threatened  by  three  enemy  land-based  anti-ship  missiles  (ASM) 
and  two  enemy  aircraft  each  loaded  with  two  ASMs.  The  AAW  frigate  has  a 
medium-range  air  defense  missile  system  (12  SAMs),  short-range  air  defense 
missile  system  (16  SAMs),  and  CIWS. 
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In  alternative  scenarios  the  number  of  land-based  ASMs  and  aircraft  are 
varied.  The  weapon  configuration  and  the  radar  range  of  the  friendly  frigate  is 
also  changed  for  each  scenario. 

After  creating  the  model  in  MANA,  a  Nearly  Orthogonal  Latin  Hypercube 
(NOLH)  design  is  used  to  design  a  set  of  experiments  by  varying  factors  of  threat 
and  weapon  capabilities.  To  analyze  the  output  of  the  experiment,  statistical 
summaries,  multiple  regression,  partition  trees,  plots,  and  graphs  are  used. 

D.  CHAPTER  OUTLINE 

Chapter  II  is  the  literature  review.  It  summarizes  the  basic  concepts  of  the 
AAW  mission  and  layered  air  defense  systems,  informs  the  reader  about  UAVs 
that  can  take  off  and  land  on  small  surface  ships  such  as  frigates  or  littoral 
combat  ships  (LCS),  and  discusses  the  agent-based  simulation  and  modeling 
tool  MANA. 

Chapter  III  contains  the  model  development  and  descriptions  of  scenarios. 
Agent  types  with  their  specifications  and  the  modeling  assumptions  are  explained 
in  this  chapter  as  well. 

Chapter  IV  discusses  the  exploration  of  the  model.  It  begins  with  an 
overview  of  the  design  of  experiment  used  for  the  simulations;  then,  it  explains 
controllable  and  uncontrollable  factors.  It  continues  with  a  detailed  analysis  of  the 
model  output  using  several  data  analysis  techniques,  such  as  least  square 
regression  and  partition  trees.  It  closes  with  a  discussion  of  the  significant  factors 
discovered  from  the  analysis. 

Chapter  V  concludes  the  thesis  with  a  summary  of  the  study, 
recommendations,  and  suggestions  for  further  research. 
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II.  LITERATURE  REVIEW 


The  traditional  approach  in  naval  warship  building  is  to  design  combat 
systems  around  the  hull  vessel  platform.  The  platform,  however,  should  be 
designed  around  the  combat  systems  to  build  more  effective  warships.  During 
the  early  conceptual  design  of  a  combat  ship,  computer  simulation  models,  and 
experimental  designs  provide  useful  insights  about  the  required  weapon 
systems,  radars  and  other  combat  systems  (MacCalman,  Beery,  &  Paulo,  2016). 
Therefore,  operational  effectiveness  becomes  independent  of  physical  design 
considerations.  In  this  study,  our  aim  is  to  evaluate  different  operational 
characteristics  of  a  frigate  in  an  AAW  environment  to  decide  the  best  mix  of 
weapons  and  sensors.  In  addition,  we  examine  contributions  of  a  prospective 
UAV  in  anti-air  warfare. 

A.  ANTI-AIR  WARFARE 

The  North  Atlantic  Treaty  Organization  (NATO)  describes  anti-air  warfare 
as  “measures  taken  to  defend  a  maritime  force  against  attacks  by  airborne 
weapons  launched  from  aircraft,  ships,  submarines  and  land-based  sites”  (Anti¬ 
air  warfare,  2015)  (Figure  3). 


According  to  Defencyclopedia,  today’s  missiles  are  in  common  use  among 
world  navies  due  to  their  long  ranges  and  accuracy  (Defencyclopedia,  2014).  The 
website  further  describes  that  most  defense  industry  companies  sell  these  deadly 
missiles  at  affordable  prices  to  allow  many  countries  to  easily  employ  them 
onboard  their  warships.  Missiles  also  offer  massive  destruction  compared  to 
large  caliber  guns  (Figure  4). 

World  navies  prefer  anti-ship  missiles  not  only  for  their  accuracy  and  long 
ranges,  but  also  for  their  simplified  launch  procedures  and  maintenance 
simplicity.  For  ships  operating  near  enemy  ASM  sites,  there  is  a  need  for  a 
defensive  system  that  can  counteract  this  powerful  threat. 


Figure  4.  Exocet  Finds  Its  Target  During  the  Falklands  War. 

Source:  Pagi  (2016). 
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Depending  on  the  speed  and  flight  paths  of  incoming  missiles,  defending 
ships  have  different  time  windows  in  which  to  acquire  the  incoming  missile  on 
their  radar  and  engage  it.  For  example,  a  missile  which  uses  a  sea  skimming 
approach  cannot  be  detected  until  it  is  20-40  seconds  from  impact 
(Defencyclopedia,  2014). 

To  defend  itself  from  any  incoming  ASM,  a  warship  is  developed  with 
multiple  layers  of  air  defenses  (Figure-5).  This  allows  engagement  of  an  inbound 
threat  by  more  than  one  weapon  system.  Layered  air  defense  usually  consists  of 
a  combination  of: 

•  Long  range  surface-to-air  missile  (SAM) 

•  Medium  range  SAM 

•  Short  range  SAM 

•  Medium  caliber  gun 

•  Point  Defense  Missile  System  (PDMS) 

•  Small  caliber  gun  with  a  high  rate  of  fire  (Close-in  Weapon  System 
-  CIWS) 

•  Electronic  jamming  and  countermeasures 

•  Passive  countermeasures  like  chaff  and  flares  (Defencyclopedia, 
2014) 
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Layered  Air  Defense  allows  engagement  of  enemy  threats  at  multiple  ranges  and 
by  more  than  one  defensive  system. 

Figure  5.  Layered  Air  Defense. 

B.  SHIP-BASED  UNMANNED  AERIAL  VEHICLES 

In  this  decade  unmanned  systems  are  very  common  in  almost  every 
environment.  They  provide  new  and  enhanced  capabilities  to  the  warfighter  (U.S. 
Department  of  Defense,  2013). 

Modern  navies  need  to  conduct  airborne  intelligence,  surveillance,  and 
reconnaissance  (ISR)  anywhere  and  anytime  (Patt,  n.d.).  The  report  also 
describes  that  current  technologies  have  their  limitations.  For  example, 
helicopters  have  limited  range  and  flight  time.  Fixed-wing  manned  and 
unmanned  aircraft  have  longer  range  but  require  longer  runways,  such  as  those 
on  aircraft  carriers  or  at  land  bases. 


8 


To  help  overcome  these  challenges  world  navies  try  to  develop  unmanned 
aerial  vehicles  that  can  be  stationed  onboard  warships,  such  as  littoral  combat 
ships,  frigates,  and  destroyers.  As  an  example,  the  U.S.  Department  of  Defense 
recently  launched  the  Tactically  Exploited  Reconnaissance  Node  (TERN) 
program  in  2014  to  fill  this  need  (Patt,  n.d.).  This  aerial  system  will  be  capable  of 
conducting  UAV  operations  from  most  ship  types  and  in  rough  sea  conditions 
without  extensive  ship  modifications. 

C.  AGENT-BASED  SIMULATIONS 

“Agent-based  simulations  are  models  where  agents,  objects,  or  entities 
sense  and  stochastically  respond  autonomously  to  conditions  in  their  local 
environments”  (Sanchez  &  Lucas,  2002).  We  use  agent-based  models  to  assess 
the  effects  of  these  individual  agents’  actions  and  interactions  on  the  system.  In 
agent-based  models  agents  behave  autonomously  according  to  predefined  rules. 

Agent-based  models  have  various  areas  of  application,  including  military 
applications.  Decision-making  processes  and  training  plans  can  be  improved  by 
using  agent-based  models.  Furthermore,  we  can  test  tactics  and  war  principles  to 
determine  better  force  structures,  decide  the  best  mix  of  weapons,  or  improve  the 
procurement  process  (Cioppa,  2003). 

There  are  many  agent-based  modeling  tools  for  military  applications. 
Several  of  these  toolkits  are  BactoWars,  EINSTein,  MASON,  NetLogo,  Repast, 
Swarm,  WISDOM,  and  MANA  (Figure  6).  MANA  is  selected  for  this  research 
because  of  its  effective  use  in  many  past  naval  studies  with  similar  research 
aims. 
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Figure  6.  Agent-Based  Simulation  of  Foraging  Sequence  for  Ants  Using 
MASON  Simulation  Toolkit.  Source:  Luke,  Cioffi-Revilla,  Panait, 
Sullivan,  &  Balan  (2016). 

D.  MAP  AWARE  NON-UNIFORM  AUTOMATA  (MANA) 

Developed  by  the  Defence  Technology  Agency  in  New  Zealand,  MANA 
Version  V  is  used  in  this  thesis  as  a  simulation  platform.  MANA  is  a  time- 
stepped,  stochastic,  agent-based  simulation  tool  designed  for  simulating  real  life 
military  scenarios  (G.  McIntosh,  2009).  (See  Figure  7). 

MANA  is  easy  to  use  and  offers  a  straightforward  interface  for  setting 
battle  parameters  (Berryman,  2008).  MANA  tries  to  capture  the  essence  of  the 
physical  and  behavioral  aspects  of  the  scenario,  but  it  avoids  unnecessary 
details  (e.g.,  detailed  flight  paths  of  missiles).  In  MANA,  a  basic  military  scenario 
can  be  built  quickly.  After  creating  a  basic  model  one  can  change  the  agent’s 
parameters  and  characteristics  to  make  the  scenario  more  accurate  and  realistic. 
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MANA  V 


DTA ** 


DEFENCE  TECHNOLOGY  AGENCY 


Vector-Based  Movement 

Map  Aware  Non-uniform  Automata 


This  software  has  been  developed  by  Gregory  McIntosh.  David  Galliqan. 

Mark  Anderson  and  Michael  Lauren  for  the  New  Zealand  Army  and  Defence  Force. 

Terms  of  use: 

This  software  belongs  to  the  Defence  Technology  Agency  (DTA),  New  Zealand 
Defence  Force.  It  is  not  to  be  distributed  without  the  permission  of  DTA.  This 
product  is  not  to  be  offered  for  resale. 


View  end  user  license  agreement  (EULA1 


See  https://teams.nzdf.mil.nz/sites/mana  for  updates  and  support. 

(Requires  user  account.) 

I  understand  I  can  only  use  this  software  subject  to  the  conditions  stated  above. 
Version  5.01.09  I  Agree  |  Cancel  | 


Figure  7.  MANA  Startup  Screen. 


The  primary  element  of  MANA  models  is  the  squad.  A  squad  can  consist 
of  one  or  more  homogeneous  agents.  Agents  in  MANA  are  map-aware,  which 
means  they  can  sense  the  characteristics  of  the  environment  through  organic 
sensors  or  inorganic  sensors  (e.g.,  communication  with  other  agents).  Each 
agent  has  different  properties.  Each  agent  behaves  independently  according  to 
the  environment  and  the  user  defined  rules. 

Each  agent  in  MANA  has  behavioral  characteristics  that  determine  its 
propensity  to  move  toward  or  away  from  particular  objects  or  agents,  such  as 
enemy  aircraft  in  AAW  missions  or  a  high  value  unit  in  military  convoy 
operations.  Users  can  change  these  settings  in  the  squad  properties  window  (G. 
McIntosh,  2007)  (Figure  8). 
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Figure  8.  Squad  Personalities  Settings  in  MANA. 


MANA  also  offers  predefined  states,  as  shown  on  the  right  side  of  the 
screenshot  in  Figure  8  (e.g.,  enemy  contact,  injured,  or  out  of  fuel)  and  a  user 
can  define  different  personality  settings  for  each  state. 

Besides  behavioral  characteristics,  each  agent  has  physical 
characteristics.  For  instance,  each  agent  has  sensors,  weapons,  communication 
links  between  each  other,  speed,  personal  concealment,  etc.  An  agent  can 
detect  and  classify  other  agents  within  range  of  its  sensors  and  can  shoot 
classified  enemy  agents  using  its  weapons  (Figure  9).  The  personal  concealment 
factor  of  an  agent  is  used  to  simulate  the  stealth  capability  of  an  agent. 
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Figure  9.  Weapon  Settings  in  MANA. 


Another  powerful  aspect  of  MANA  is  that  agents  can  share  situational 
awareness  through  communication  links.  Users  can  determine  the  characteristics 
of  these  communication  links,  such  as  range,  reliability,  delivery  method 
(guaranteed  delivery  or  fire  and  forget),  and  communication  link  latency  (G. 
McIntosh,  2007). 

MANA  is  a  commonly  used  simulation  tool  at  the  Naval  Postgraduate 
School  (NPS),  and  it  has  been  widely  used  for  both  military  and  academic 
studies  by  NPS  faculty  and  students.  Past  studies  similar  to  our  approach  include 
UAV  contributions  to  expeditionary  operations  (Raffetto,  2004),  counter-piracy 
operations  in  the  Gulf  of  Aden  (Tsilis,  2011),  operational  effectiveness  of  a  small 

surface  combat  ship  in  an  anti-surface  warfare  environment  (Kaymal,  2013),  and 
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effectiveness  of  unmanned  surface  vehicles  in  anti-submarine  warfare  (Unlu, 
2015).  In  this  study  MANA  is  used  in  a  similar  manner  for  simulating  tactical 
situations,  but  unlike  the  studies  mentioned  previously,  the  focus  is  an  AAW 
environment.  In  addition,  the  purpose  of  this  study  is  to  inform  ship  design,  not 
necessarily  specific  tactical  employment  of  systems  or  people. 
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III.  MODEL  DEVELOPMENT 


A.  SCENARIO  DESCRIPTION 

In  this  thesis  the  agent-based  combat  modeling  tool  MANA  is  used  to 
simulate  a  single  frigate  in  an  anti-air  warfare  environment. 

In  the  scenario  there  is  one  friendly  AAW  frigate,  and  it  has  one  UAV 
onboard.  The  frigate  is  attacked  by  both  land  based  anti-ship  cruise  missiles 
(ASMs)  and  air-to-surface  ASMs  launched  by  enemy  aircraft.  The  frigate  uses  its 
layered  air  defense  system  to  protect  against  these  enemy  assets.  A  screenshot 
of  the  scenario  from  MANA  appears  in  Figure  10. 


Figure  1 0.  Screenshot  of  the  Tactical  Scenario. 
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The  battlefield  is  150  x  100  nautical  miles  (nm).  In  the  MANA  scenario 
(Figure  10),  the  blue  colored  area  represents  the  sea  and  the  yellow  colored  area 
is  land. 

B.  AGENT  DESCRIPTIONS 

There  are  five  different  types  of  agents  in  the  scenario.  A  frigate  and  a 
UAV  onboard  the  frigate  are  friendly  assets.  Enemy  assets  are  land-based 
ASMs,  aircraft,  and  air-to-surface  ASMs  launched  by  the  aircraft. 

1.  Friendly  Assets 

Friendly  assets  are  trying  to  defend  themselves  while  conducting  an  ISR 
mission  in  the  area  of  interest.  The  UAV  shares  its  situational  awareness  with  the 
frigate  through  a  directed  data  link. 

a.  Frigate 

In  this  scenario  the  AAW  frigate  moves  in  a  predetermined  course  with  a 
speed  of  25  knots  (kts).  The  frigate  has  sensors  and  weapons  to  detect  and 
neutralize  enemy  assets.  If  the  frigate  detects  any  enemy  agent,  it  changes 
course  to  close  with  them. 

There  are  two  types  of  sensors  onboard  the  frigate.  One  of  them  is  a 
general  purpose  air-search  radar,  which  is  used  to  detect  aircraft.  The  other  is  a 
missile-search  radar,  which  is  good  at  detecting  relatively  small  targets  but  has 
shorter  range.  Table  1  summarizes  the  characteristics  of  these  radars. 


Table  1 .  Characteristics  of  Sensors  Onboard  the  Frigate. 


Radar 

Range 

Target  Type 

Air  search  radar 

53,000*  m 

Aircraft 

Missile  search  radar 

25,000  m 

Missiles 

*Air  search  radar  range  is  changed  in  the  design  to  find  the  best  design  range. 
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The  frigate’s  layered  defense  is  limited  to  four  weapon  systems  due  to 
space  restriction.  A  gun  system  is  always  preferred  to  be  onboard.  The  other 
three  weapon  systems  consist  of  two  surface-to-air  missile  systems  (SAMs)  and 
either  a  Close-in  Weapon  System  Gatling  gun  (CIWS)  or  a  Point  Defense  Missile 
System  (PDMS).  These  three  weapon  systems  are  changed  in  the  experimental 
design  to  determine  the  best  mix  of  weapon  types  for  an  AAW  frigate  in  this 
tactical  scenario.  Table  2  summarizes  the  weapon  types  and  their  specifications. 
P(Hit)  represents  the  probability  of  hit  and  MANA  interpolates  this  value  for  any 
distance  between  the  maximum  and  minimum  range. 

Table  2.  Weapon  Types  and  Specifications. 


Weapon 

Minimum  Range 

/  P  (Hit) 

Maximum  Range 

/  P  (Hit) 

Target  Type 

CIWS 

1,000  m/  0.6 

6,000  m/  0.4 

Only  missiles 

PDMS 

1,000  m/  0.8 

9,000  m/  0.6 

Only  missiles 

Gun 

8,000  m/  0.3 

14,000  m/  0.2 

Both  aircraft  and 

missiles 

Short  Range  SAM 

5,000  m/  0.8 

30,000  m/  0.3 

Both  aircraft  and 

missiles 

Medium  Range 

SAM 

10,000  m/  0.8 

70,000  m/  0.1 

Both  aircraft  and 

missiles 

Long  Range  SAM 

10,000  m/  0.8 

200,000  m/  0.1 

Both  aircraft  and 

missiles 
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For  simulation  and  experimental  design  purposes,  we  categorize  SAM  and 
CIWS/PDMS  alternatives  into  options.  Table  3  depicts  the  mapping  between  the 
option  number  and  the  actual  weapon  names  in  that  option.  Again,  a  main  gun 
system  is  present  for  every  option. 


Table  3.  Weapon  Selection  Options. 


SAM  Options 

Active  SAM  Names 

1 

Medium  Range  SAM  &  Short  Range  SAM 

2 

Long  Range  SAM  &  Short  Range  SAM 

3 

Long  Range  SAM  &  Medium  Range  SAM 

CIWS/PDMS  Options 

Active  Weapon  (CIWS  or  PDMS) 

1 

CIWS 

2 

PDMS 

b.  Ship-based  UAV 

The  ship-based  UAV’s  aim  is  to  provide  early  warning  for  the  frigate  via  a 
data  link.  The  UAV  flies  in  a  predetermined  course  and  tries  to  detect  enemy 
assets  with  its  cookie-cutter  sensor,  which  has  range  of  60,000  meters.  A  cookie- 
cutter  sensor  means  that  it  detects  any  target  within  its  range  with  certainty 
(Figure  11).  The  UAV  does  not  have  any  weapon.  For  each  tactical  situation 
simulated  with  a  UAV,  the  aircraft  is  assumed  airborne  at  the  time  of  the  attack. 
We  did  not  assign  an  on  station  time  for  the  UAV  because  each  attack  takes  at 
most  30  minutes.  Table  4  summarizes  the  capabilities  of  the  UAV. 
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Table  4.  UAV  Capabilities. 


Sensor 

Weapon 

Speed 

Data  Link 

Cookie  cutter  60,000  m 

Not  Applicable 

1 50  kts 

Yes.  360,000  m 

P(Detection) 


Figure  1 1 .  Graphical  Explanation  of  a  Cookie-Cutter  Sensor. 

2.  Enemy  Assets 

Enemy  assets  consist  of  land-based  ASMs,  aircraft,  and  air-to-surface 
ASMs  launched  by  aircraft. 

a.  Land-based  ASMs 

Land-based  ASMs  proceed  directly  to  the  target  frigate  using  their  inertial 
navigation  system  and  active  radar.  They  explode  within  100  meters  of  the 
target.  The  specifications  of  the  ASMs  are  as  shown  in  Table  5. 


Table  5.  Land-based  ASMs  Specifications. 


Enemy  Name 

Range 

Speed 

Guidance  System 

Land-based  ASM 

180  nm 

1 ,800  kts 

Active  Radar  Homing 

b.  Aircraft 

Aircraft  are  each  equipped  with  two  air-to-surface  ASMs.  They  launch  both 
missiles  when  the  friendly  AAW  frigate  is  in  range  of  their  weapon.  After  firing 
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their  missiles,  they  fly  back  to  their  base  to  avoid  any  possible  enemy  fire.  We  do 
not  determine  flight  time  limits  for  the  aircraft  because  the  tactical  simulation 
takes  only  30  minutes  and  is  close  to  shore,  well  within  the  range  of  most  tactical 
aircraft.  Specifications  of  the  enemy  aircraft  are  as  shown  in  Table  6. 


Table  6.  Enemy  Aircrafts  Specifications. 


Enemy  Name 

Sensors 

Weapons 

Speed 

Aircraft 

Cookie-cutter 
50,000  m 

2  air-to-surface 
ASMs 

300  kts 

c.  Air-to-Surface  ASMs 

ASM  missiles  are  launched  from  enemy  aircraft  and  have  an  active  radar 
guidance  system.  They  fly  directly  to  the  target  with  a  speed  of  1 ,800  knots.  They 
explode  when  they  reach  500  meters  of  the  target.  Specifications  of  the  air-to- 
surface  missiles  appear  in  Table  7. 


Table  7.  Enemy  Air-to-Surface  ASMs  Specifications. 


Enemy  Name 

Range 

Speed 

Guidance  System 

Air-to-Surface  ASMs 

50,000  m 

1 ,800  kts. 

Active  Radar  Homing 

C.  STOP  CONDITIONS 

In  MANA,  stop  conditions  cause  the  simulation  to  terminate  in  order  to 
reduce  run  time  and  save  overall  experiment  time  (Figure  12).  The  tactical 
simulation  for  our  model  will  stop  when  either  or  both  of  the  following  conditions 
occur: 

•  The  friendly  frigate  is  killed. 

•  All  enemy  assets  are  neutralized  by  the  friendly  frigate. 
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#  StopC...  -  □  X 

Stop  When: 

□  Red  Reach  Final  Goal 

□  Blue  Reach  Final  Goal 

□  Neutral  Reach  Final  Goal 

□  Red  Losses  = 

□  Blue  Losses  = 

□  Neutral  Losses  =[T 


□  Squad  [i  HI  [^hed 

□  Squad  Losses  = 

for  Squad 


✓  OK 


X  Cancel 


Figure  12.  Stop  Conditions  Menu  in  MANA. 


D.  SCENARIO  ASSUMPTIONS  AND  LIMITATIONS 

Any  simulation  includes  assumptions  and  limitations  because  it  is 
impossible  to  imitate  exact  real-world  events.  These  assumptions  and  limitations, 
however,  need  to  be  plausible  to  provide  useful  insights  about  the  real  world. 

1.  Assumptions  and  Constraints 

The  key  assumptions  and  constraints  for  this  analysis  are: 

•  Only  air  threats  are  taken  into  account. 

•  Aircraft  fly  back  to  their  base  as  soon  as  they  launch  missiles. 

•  Logistics  are  not  considered  (no  reload  of  weapons  for  each  side). 

•  Two  missiles  are  loaded  on  each  enemy  aircraft. 

•  Space  in  the  frigate  design  limits  the  number  of  AAW  defense 
systems  to  three  (besides  medium  caliber  gun).  Soft  kill  methods 
are  neglected  (electronic  jamming,  chaff,  and  flares). 

•  UAV  shares  its  situational  awareness  with  the  frigate. 
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2. 


Limitations 


Limitations  are  related  to  the  modeling  platform,  MANA,  and  the 
unclassified  information  sources. 

•  Performance  characteristics  of  radars  and  weapons  are  derived 
from  open  internet  sources.  Therefore,  they  are  not  exact. 

•  Many  features  of  aerial  platforms  and  weapons  are  not  included  in 
MANA.  For  example: 

1 .  An  actual  weaving  flight  path  of  a  misslie  cannot  be 
simulated. 

2.  Probability  of  detection  does  not  differ  according  to  the 
aircraft  or  missile’s  altitude. 

E.  MEASURE  OF  EFFECTIVENESS 

Survivability  of  friendly  frigate  and  the  number  of  red  casualties  are  the 
measures  of  effectiveness  (MOEs)  in  this  study.  In  this  analysis,  the  frigate 
survives  if  it  is  hit  two  or  fewer  times.  If  the  frigate  survives  after  all  enemy 
attacks,  it  is  a  success.  If  the  frigate  is  shot  three  or  more  times  and  killed,  it  is  a 
failure. 

For  the  second  MOE  we  focus  on  the  number  of  enemy  casualties,  either 
aircraft  or  missile. 
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IV.  MODEL  EXPLORATION 


A.  DESIGN  OF  EXPERIMENTS 

The  design  of  experiment  (DOE)  is  a  technique  to  examine  the 
relationships  between  design  factors  and  outcomes.  Design  factors  are  the 
inputs  to  the  simulation,  and  the  outcomes  are  the  two  MOEs. 

To  create  a  simulation  of  military  operations,  the  designer  must  consider 
many  factors  that  affect  the  outcome.  These  factors  can  be  controllable  or 
uncontrollable.  For  example,  weapon  selection  onboard  a  frigate  is  a  controllable 
factor,  but  the  number  of  enemy  aircraft  is  an  uncontrollable  factor. 

In  this  thesis’  experimental  design,  there  are  seven  controllable  and  two 
uncontrollable  factors.  These  factors  are  discussed  in  the  subsequent  section. 
The  nearly  orthogonal  Latin  hypercube  (NOLH)  spreadsheet  is  used  to  generate 
design  points  in  this  analysis  (Sanchez,  2011).  NOLH  designs  have  good  space¬ 
filling  properties,  and  they  are  almost  orthogonal  (Cioppa  &  Lucas,  2007).  For 
example,  the  maximum  correlation  between  the  columns  in  our  design  matrix 
used  in  this  thesis  is  0.0659  (Figure  13).  The  scatterplot  matrix  of  the  NOLH 
design  we  used  appears  in  Figure  14.  For  greater  flexibility  in  generating  NOLHs, 
see  Hernandez  et  al.  2012. 


Correlations 

SAM  Option  CIWS/PDMS  UAV  BlueShipRadar  LandBasedASMs  RedAircrafts  BlueShipShortSAMRng  BlueShipMedSAMRng  BlueShipLongSAMRng 


SAM  Option 

1.0000 

0.0165 

0.0274 

0.0078 

-0.0403 

0.0029 

0.0481 

-0.0030 

-0.0625 

CIWS/PDMS 

0.0165 

1.0000 

0.0039 

0.0608 

-0.0064 

-0.0061 

0.0478 

0.0151 

-0.0157 

UAV 

0.0274 

0.0039 

1.0000 

-0.0556 

-0.0064 

0.0101 

0.0090 

-0.0531 

-0.0203 

BlueShipRadar 

0.0078 

0.0608 

-0.0556 

1.0000 

0.0207 

0.0291 

0.0000 

0.0000 

0.0005 

LandBasedASMs 

-0.0403 

-0.0064 

-0.0064 

0.0207 

1.0000 

-0.0496 

-0.0087 

0.0093 

-0.0196 

RedAircrafts 

0.0029 

-0.0061 

0.0101 

0.0291 

-0.0496 

1.0000 

-0.0233 

-0.0586 

0.0659 

BlueShipShortSAMRng 

0.0481 

0.0478 

0.0090 

0.0000 

-0.0087 

-0.0233 

1.0000 

-0.0000 

0.0015 

BlueShipMedSAMRng 

-0.0030 

0.0151 

-0.0531 

0.0000 

0.0093 

-0.0586 

-0.0000 

1.0000 

0.0009 

BlueShipLongSAMRng 

1.0000 

Figure  13.  Correlation  Matrix  of  the  Factors. 
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▼  Scatterplot  Matrix 
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The  space  filling  property  of  the  NOLH  design  can  be  easily  seen  in  the  above 
plot. 


Figure  1 4.  Scatterplot  Matrix  for  the  Factors. 

B.  DESIGN  FACTORS 

Many  factors  can  affect  an  AAW  operation.  In  the  simulation,  a  total  of 
nine  factors  were  varied  to  determine  the  best  combination  of  weapons.  A  list  of 
factors  and  their  explanation  is  shown  in  Table  8.  These  factors  are  explained  in 
the  subsequent  sections  as  well. 
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Table  8.  Design  Factors. 


Factor  Name 

Explanation 

Min 

Max 

Unit 

Controllable  Factors 

SAM  Option 

Selection  of  surface  to 

air  missiles  onboard 

Refer  to  Table  3 

CIWS/PDMS 

Selection  of  either 

CIWSorPDMS 

Refer  to  Table  3 

UAV 

Presence  of  UAV 

0 

1 

- 

BlueShipRadar 

Range  of  the  frigate’s 

air  search  radar 

40,000 

250,000 

meter 

BlueShipShortSAMRng 

Range  of  the  frigate’s 

Short  range  SAM 

5,000 

35,000 

meter 

BlueShipMedSAMRng 

Range  of  the  frigate’s 

Short  range  SAM 

10,000 

85,000 

meter 

BlueShipLongSAMRng 

Range  of  the  frigate’s 

Short  range  SAM 

10,000 

200,000 

meter 

Uncontrollable  Factors 

LandBasedASMs 

Number  of  enemy 

land  based  anti-ship 

missiles 

2 

6 

- 

RedAircraft 

Number  of  enemy 

aircraft  (each  carrying 

two  ASMs) 

2 

5 

- 
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1. 


Controllable  Factors 


Controllable  factors  can  be  determined  during  the  ship  design  process  of 
the  frigate.  In  this  thesis,  selection  of  the  SAM  types  onboard,  CIWS  or  PDMS 
selection,  presence  of  UAV  and  range  of  the  sensors  and  the  SAMs  are 
controllable  factors. 

a.  Surface-to-Air  Missile  Options 

As  mentioned  earlier,  we  divided  SAMs  into  three  groups:  short  range 
SAM,  medium  range  SAM,  and  long  range  SAM.  Due  to  the  space  restriction  in 
the  frigate,  we  employ  two  SAM  types  at  a  time.  The  possible  selections  of  these 
two  missile  types  and  number  of  available  missiles  onbaord  are  as  shown  in 
Table  9. 


Table  9.  Possible  Selection  of  SAMs 


SAM  Options 

Active  SAM  Names 

1 

Medium  Range  SAM  (12  missiles)  &  Short 

Range  SAM  (16  missiles) 

2 

Long  Range  SAM  (12  missiles)  &  Short 

Range  SAM  (16  missiles) 

3 

Long  Range  SAM  (12  missiles)  &  Medium 

Range  SAM  (16  missiles) 

b.  CIWS  versus  PDMS  Selection 

To  evaluate  the  effectiveness  of  the  CIWS  and  PDMS  in  an  AAW  mission, 
we  introduce  a  factor  which  activates  and  deactivates  either  CIWS  or  PDMS 
onboard  the  frigate.  In  doing  so,  we  intend  to  find  which  weapon  type  is  superior 
to  the  other.  Table  10  explains  the  mapping  between  selection  number  and  the 
active  weapon  system. 
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Table  10.  CIWS  or  PDMS  Selection 


CIWS/PDMS  Options 

Active  Weapon  (CIWS  or  PDMS) 

1 

CIWS  (12  burst) 

2 

PDMS  (21  missiles) 

It  is  very  easy  to  simulate  most  weapons  in  MANA  due  to  its  user  friendly 
interface.  Simulating  rapid  firing  guns  like  CIWS,  however,  is  not  straightforward. 

In  MANA  the  weapon  properties  menu  is  designed  for  missiles,  because  it 
asks  for  a  set  of  values  for  a  single  shot  rather  than  a  burst  of  fire.  For  example, 
MANA  uses  a  number  of  shots  per  second  and  calculates  the  remaining 
ammunition  according  to  this  ratio.  To  overcome  this  issue  we  input  the  values  of 
the  CIWS  burst  capacity  rather  than  an  individual  round. 

c.  Presence  of  UA  V 

Whether  a  UAV  is  present  in  the  AAW  scenario  is  a  controllable  factor  in 
the  experimental  design.  This  factor  can  represent  the  capability  of  the  frigate 
and  it  may  be  related  to  the  tactics  as  well.  Even  if  the  frigate  is  capable  of 
carrying  a  UAV,  the  commanding  officer  may  choose  not  to  conduct  UAV 
operations  due  to  the  tactical  situation. 

d.  Sensor  Range 

In  AAW  operations,  the  frigate  has  to  detect  and  classify  the  enemy  assets 
as  soon  as  possible  to  defend  effectively.  Classification  of  the  enemy  assets  is 
crucial  in  defense  and  depends  on  design  characteristics  of  the  ship’s  radar  and 
time  to  classify  once  a  target  is  detected.  Therefore,  the  radar’s  range  is 
designed  to  be  a  controllable  factor  in  the  study.  It  varies  from  40,000  meters  to 
250,000  meters. 
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e.  Surface-to-Air  Missile  Ranges 

There  are  different  types  of  SAMs  in  the  defense  industry  and  they  differ 
from  each  other  in  their  capabilities  and  costs.  To  see  the  effect  of  different 
SAMs’  ranges  in  an  AAW  engagement,  this  is  introduced  in  the  experiment  as  a 
controllable  factor. 

2.  Uncontrollable  Factors 

Uncontrollable  factors,  also  known  as  noise  factors,  are  related  to  enemy 
capabilities  or  characteristics  of  the  operational  environment.  In  this  thesis,  there 
are  two  uncontrollable  factors:  number  of  enemy  aircraft  (loaded  with  two  ASMs) 
and  number  of  enemy  land-based  ASMs. 

a.  Enemy  Aircraft 

The  number  of  enemy  aircraft  is  a  factor  in  the  experimental  design.  It 
ranges  from  two  to  five  aircraft.  Each  aircraft  launches  its  ASMs  individually  as 
soon  as  the  frigate  is  in  range  of  their  weapon,  and  each  aircraft  flies  back  to  its 
base. 


b.  Enemy  Land-based  ASMs 

The  number  of  enemy  land-based  ASMs  is  another  factor  that  ranges  from 
two  to  six  missiles. 

C.  DATA  ANALYSIS 

In  this  section  we  explain  the  analysis  tool  used,  model  runs,  and  initial 
analysis  of  the  data,  and  then  we  discuss  regression  analysis  and  partition  tree 
analysis. 

1.  Analysis  Tool 

JMP  statistical  discovery  software  offers  both  powerful  statistics  and 
dynamic  graphics  capabilities  to  its  users.  In  the  thesis,  JMP  statistical  discovery 
software  version  12.0.1  is  used  to  analyze  the  data. 
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2. 


Model  Runs 


As  previously  indicated,  nine  factors  are  included  in  the  model.  Two 
hundred  fifty-seven  design  points  were  generated  using  the  NOLH  design  and 
100  replications  for  each  of  these  design  points  were  simulated.  As  a  result 
25,700  AAW  scenarios  are  simulated. 

In  this  model,  one  second  is  one  simulated  time  step.  As  Buss  and  Al 
Rowaei  state,  the  time  step  selection  has  an  important  impact  on  the  outcome  of 
the  model  (Buss  &  Al  Rowaei,  2010).  They  also  explain  that  models  with  larger 
time  steps  take  less  time  but  may  yield  erroneous  results.  Therefore,  time  step 
selection  should  be  considered  thoroughly  to  get  correct  results  in  a  reasonable 
time  period.  Because  of  the  high  speed  of  ASMs,  a  one-second  time  step  is 
selected  to  capture  the  rapid  nature  of  AAW  tactics. 

3.  Initial  Analysis  of  the  Data 

A  total  of  25,700  rows  of  raw  data  from  the  simulation  experiments  are 
imported  into  JMP  for  analysis.  To  explore  the  survivability  of  the  frigate,  we 
created  a  new  column  by  subtracting  the  frigate  casualty  percent  column  from 
one.  Figure  15  displays  the  distribution  of  the  frigate’s  probability  of  survival  in 
the  overall  replications.  Average  survivability  is  around  0.551,  with  a  standard 
deviation  of  0.497.  The  upper  and  lower  95  percent  confidence  interval  is  as 
shown  in  Figure  15. 
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▼  Frigate  survivability 
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^3  r  Summary  Statistics 
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Std  Dev  0.4973364 

Std  ErrMean _ 0.00310B 

f0pper95%  Mean  0.55771491 
Lower  95%  Mean  0.5455536J 
N~“  25700 


) 


The  right  bar  in  the  figure  corresponds  to  the  proportion  of  the  runs  in  which  the 
frigate  survives  (55%  of  the  time)  and  the  left  bar  the  proportion  in  which  it  does 
not  (45%  of  the  time). 


Figure  15.  Distribution  for  the  Mean  Frigate  Survivability. 


For  the  exploration  of  the  second  MOE,  we  created  a  distribution  plot  of 
the  mean  number  of  enemy  casualties  (Figure  16).  As  it  appears  in  Figure  16, 
the  mean  enemy  casualties  has  a  bimodal  distribution  with  a  mean  of  8.748 
casualties  and  a  standard  deviation  of  4.745.  The  distribution  has  two  distinct 
peak  points,  in  other  words,  two  modes. 
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Figure  16.  Distribution  for  the  Mean  Enemy  Casualties  Showing  a  Bi¬ 
modal  Characteristic. 
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4.  Regression  Analysis 

In  our  raw  data  there  are  100  replications  for  each  of  the  257  design 
points.  Although  the  input  parameters  stay  the  same  across  100  replications,  the 
outcome  will  vary  due  to  stochasticity  originating  from  the  MANA  modeling 
platform.  This  randomness  causes  difficulty  in  fitting  the  regression  model.  To 
overcome  this  difficulty,  we  summarize  the  data  by  taking  the  means  of  the 
factors  and  the  response  for  each  design  point.  Therefore,  the  survivability 
response  becomes  the  probability  of  survivability  ranging  from  zero  to  one,  and 
the  number  of  enemy  casualties  becomes  the  mean  number  of  enemy  casualties 
ranging  from  zero  to  19.  They  are  now  both  continuous  variables. 

Distributions  of  the  frigate’s  survivability  and  the  enemy  casualties  for  the 
summarized  data  are  as  shown  on  Figure  17  and  Figure  18.  We  should  note  that 
the  summary  statistics  are  the  same  as  the  previous  distributions. 
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Figure  17.  Distribution  for  the  Mean  Frigate  Survivability  (summarized 

data). 


31 


▼  Mean(Alleg2Cas(Red)) 
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Figure  18.  Distribution  for  the  Mean  Enemy  Casualties  (Summarized 

Data). 


We  use  regression  analysis  to  investigate  the  relationships  between  the 
design  factors  and  our  MOEs.  There  are  many  techniques  for  regression 
analysis;  the  corrected  Akeike  Information  Criterion  (AlCc)  stepwise  technique  is 
used  in  this  study. 

We  first  fit  the  model  with  only  main  effect  terms,  then  we  add  a  second 
order  polynomial  and  two-way  interaction  terms  into  the  model  to  investigate  their 
value  as  predictors. 

We  need  to  note  that,  due  to  the  binary  responses,  the  basic  assumptions 
for  linear  regression  are  not  met.  Specifically,  the  errors  will  not  be  normally 
distributed  with  a  constant  variance.  In  addition,  the  regression  equation  may 
make  predictions  less  than  zero  or  more  than  one  at  the  extremes,  while  the 
response  must  be  restricted  to  between  zero  and  one. 

However,  the  purpose  of  our  analysis  is  to  identify  and  quantify  the 
relationship  between  the  input  variables  and  the  response  rather  than  predicting 
the  response  (Kleijnen  et  al.  2005).  As  noted  by  Hellevik  (2009),  “the  intuitively 
meaningful  interpretation  of  linear  regression  makes  it  easier  to  communicate 
research  results  than  logistic  regression.”  The  p-values  cannot  be  reliably  used 
because  the  error  terms  are  not  normally  distributed.  The  coefficient  estimates 
are  not  optimal  in  terms  of  power,  but  they  are  unbiased.  Moreover,  optimal 


32 


power  is  not  a  critical  issue  given  our  large  sample  and  high  R-square  value.  For 
more  discussion  on  this,  see  Hellevik  (2009). 

a.  Main  Effects  Model  for  the  Frigate  Survivability 

Using  the  stepwise  method  with  only  main  effect  terms  we  defined  the 
best  predictors  as  shown  in  Figure  19.  The  factors  are  listed  in  order  according  to 
their  effects  on  the  response.  For  example,  the  effect  of  the  CIWS/PDMS 
selection  on  survivability  is  greater  than  the  range  of  the  radars.  We  should  also 
note  the  two  uncontrollable  factors  of  number  of  red  aircraft  and  land  launched 
ASCMs  are  in  the  model.  The  UAV  factor,  though,  is  not  included  in  the  model  as 
a  significant  predictor  of  survivability. 
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|  LandBasedASMs'A' 

I  SAM  Option 
I  BlueShipMedSAMRng 
I  BlueShipRadarRng 
|  BlueShipShortSAMRng 


PValue 

0.00000 
0.00000 
0.00000 
0.00000 
0.00000 
0.00580 
0.03662 
0.13803  < 


★  Uncontrollable  factors 


Figure  1 9.  Effect  Summary  of  the  Factors  for  Main  Effects  Model. 


Figure  20  displays  the  actual  values  by  predicted  values  and  summary  of 
the  fitted  model.  As  mentioned  earlier,  the  UAV  factor  is  excluded  from  the 
model,  because  it  is  not  significant  enough  to  enter  the  model.  The  R-Square 
value  of  the  model  is  0.857,  which  means  our  model  can  explain  85.7  percent  of 
the  variability  of  the  response. 
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Actual  by  Predicted  Plot 


-02  0  02  0 A  0.6  OB  1  1.2  14  Root  Mean  Square  Error  0.136946 

Mean(Frigate  survivability)  Predicted  P<.0001  Mean  of  Response  0.55 1634 

RSq=0.86  RMSE=0.1369  Observations  (or  Sum  Wgts)  257 


Figure  20.  Actual  by  Predicted  Plot  and  the  Summary  of  Fit  for  the  Main 

Effects  Model. 


Figure  21  displays  the  expanded  parameter  estimates  for  the  main  effects 
model.  The  t  ratio  values  represent  the  significance  of  the  factors’  effect  on 
survivability.  Factors  indicated  with  red  dashed  boxes  have  more  significant 
effect  on  survivability  as  compared  to  other  factors. 


Expanded  Estimates 

Nominal  factors  expanded  to  all  levels 

Term 

Estimate 

Std  Error 

t  Ratio 

Prob>|t| 

lnterce£t_ 

jciWS/PDMSm] 

0.8066803 

0.057718 

13.98 

<.0001* 

-0.294794 

0.008572 

[”34,39™ 

<.0001* 

[ciWS/PDMS[2]_| 

0.2947938 

0.008572 

1  34.39J 

<.0001* 

BlueShipRadarRng 

2.9572e-7 

1.407e-7 

““To"’ 

0.0366* 

jTan as  e3AST?l  j 
|RedAir  crafty 

-0.064224 

0.00697 

r---; 

<.0001* 

-0.088704 

0.008963 

l_-9.9pj 

<.0001* 

BlueShipShortSAMRng 

1 .4659e-6 

9.852e-7 

”49 

0.1380 

B 1  u  eShip  M  edSAM  Rnq 

1.0959e-6 

3.938e-7 

^2.78^ 

0.0058* 

jBlueShipLongSAMRngj 

1.6967e-6 

1.558e-7 

[l"89"i 

<.0001* 

(SAM  Ogtion[1^ _ 

-0.098369 

0.013407 

LLMJ 

<.0001* 

SAM  Option[2] 

0.0369359 

0.011393 

3.24 

0.0014* 

SAM  Option[3] 

0.061433 

0.013354 

4.60 

<.0001* 

Figure  21 .  Expanded  Estimates  for  the  Main  Effects  Model. 
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As  shown  in  the  prediction  expression  (Figure  20),  an  increase  in  the 
uncontrollable  factors  decreases  the  probability  of  survivability.  Conversely,  an 
increase  in  controllable  factors  increases  the  survivability  rate.  Interestingly,  the 
selection  of  the  CIWS  (CIWS/PDMS=1)  and  SAM  Option  1  (Medium  and  short 
range  SAM)  decreases  the  probability  of  survival. 


Prediction  Expression 

0.80668032351503 


+  Match)  CIWS/PDMS] 


1 

2 


=>  -0.29479381 19832 
=>  0.29479381 198324 


else=> . 


+  0.00000029572071  *  BlueShipRadarftng 
+  -0.0642238997863*  LandBasedASMs 
+  -0.0887042504378*  RedAircrafts 
+  0.00000146588744*  BlueShipShortSAMRng 
+  0.00000109588815*  BlueShipMedSAMRng 
+  0.00000169667207*  BlueShipLongSAMRng 

1  =>  -0.098368842621 

2  =>  0.03693587998977 

3  =>  0.061432962631 25 

else=> . 


+  Match)  SAM  Option] 


Figure  22.  Prediction  Expression  of  the  Main  Effects  Model. 


As  a  result  of  this  analysis,  we  conclude  that  selection  of  PDMS 
(CIWS/PDMS=2),  a  mix  of  medium  and  long  range  SAM  (SAM  Option=3),  and 
longer  ranges  in  radar  and  missiles  are  recommended  for  a  better  survivability 
rate. 
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b.  Second  Order  Model  for  Frigate  Survivability 

After  exploring  the  main  effects  model,  we  fit  the  model  with  main  effects, 
second  order  polynomial,  and  two-way  interaction  terms  as  predictors,  to  find 
their  impact  on  the  frigate’s  survival  probability.  The  terms  that  are  significant 
enough  to  enter  the  model  appear  in  Figure  23.  The  first  five  significant  factors 
are  still  main  effects,  although  we  added  polynomials  and  the  two-way 
interactions  to  the  model. 


Source  LogWorth 

jciWS/PDMS  “l  04*795" 

fBlueShiplongSAMRng  26.945 

RedAircrafts  23.411 

LandBasedASMs  20.918 

pAM  Option  11.682 

cl\VS''PDMS*BlueShTpTongSAMRng  “06 

CIWS/PDMS*LandBasedASMs  4.570 

BlueShipRadarRng’BlueShipRadarRng  3.656 

CIWS/PDMS*RedAirc  rafts  2.910 

BlueShipMedSAMRng  2.850 

BlueShipRadarRng'RedAircrafts  2.416 

LandBasedASMs’RedAircrafts  2.004 

BlueShipRadarRng  1 .858 

LandBased  ASMs'BlueShipLongSAMRng  1 .726 

BlueShipMedSAMRng'BlueShipLongSAMRng  1 .095 

BlueShipShortSAMRng  1.043 

LandBasedASMs’BlueShipMedSAMRng  0.855 

UAV  0.413 

UAVRedAircrafts  0.306 

BlueShipShortSAMRng’BlueShipMedSAMRng  0.167 


PValue 


ro“ooooo] 
0.000001 
0.00000 1 
o.oooooj 
I  o.oooooj 
| 0.00001 
0.00003 
0.00022 
0.00123 
0.00141 
0.00384 
0.00991 
0.01387 
0.01879 
0.08037 
0.09067 
0.13972 
0.38638 
0.49379 
0.68040 


A 


Figure  23.  Effect  Summary  of  the  Factors  for  the  Second  Order  Model. 


The  actual  by  predicted  plot  and  the  summary  of  fit  for  the  second  order 
model  is  as  shown  in  Figure  24.  The  R-squared  value  of  this  model  is  .90,  which 
means  that  90  percent  of  the  variability  in  the  simulation  can  be  explained  using 
this  regression  model. 
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Actual  by  Predicted  Plot 


Summary  of  Fit 

RSquare  0.901292 

RSquare  Adj  0.892472 

Root  Mean  Square  Error  0.1 1 9467 

Mean  of  Response  0.551634 

Observations  (or  Sum  Wgts)  257 

Figure  24.  Actual  by  Predicted  Plot  and  the  Summary  of  Fit  for  the 

Second  Order  Model. 

Expanded  parameter  estimates  for  the  main  effects  model  are  shown  in 
Figure  25.  According  to  the  t  ratio  values,  the  same  main  effect  terms  as  in  the 
previous  model  are  more  significant  in  regard  to  survivability  as  compared  to  the 
others.  The  two-way  interactions — CIWS/PDMS  x  Land-based  ASM  and 
CIWS/PDMS  x  range  of  the  long  range  SAM — are  relatively  significant  factors 
compared  to  other  two-way  interactions.  Interestingly,  the  quadratic  term  of  the 
radar  range  has  a  greater  effect  than  the  main  effect  of  the  radar  range. 
Therefore,  changes  in  the  radar  range  affects  the  frigate’s  survivability  non- 
linearly. 
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Expanded  Estimates 


Nominal  factors  expanded  to  all  levels 


Term 

Estimate 

Std  Error 

t  Ratio 

Prob>  1 1 1 

Intercept 

0.8440495 

0.05131 

16.45 

o 

o 

Q 

CIWS/PDMS(1] 

-0.294669 

0.007478 

-3940 

<.0001* 

CIWS/PDMS[2] 

0.2946688 

0.007478 

3940 

<0001* 

UAV[0] 

-0.006491 

0.00748 

-0.87 

0.3864 

UAV[1] 

0.0064911 

0.00748 

0.87 

0.3864 

BlueShipRadarRng 

3.0489e-7 

1.23e-7 

2.48 

0.0139* 

LandBasedASMs 

-0.064341 

0.006082 

-10.58 

<.0001* 

RedAircrafts 

-0.088864 

0.00782 

-1136 

<.0001* 

BlueShipShortSAM  Rng 

1.4601e-6 

8.595e-7 

1.70 

0.0907 

BlueShipMedSAMRng 

1.1 1 13e-6 

3.44e-7 

3.23 

0.0014* 

BlueShipLongSAMRng 

1.6933e-6 

1.36e-7 

12.45 

<.0001* 

CIWS/PDMS(1]*(landBasedASMs-4.00778) 

-0.026331 

0.006148 

-4.28 

<.0001* 

CIWS/PDMS(2]*(LandBasedASMs-4.00778) 

0.0263314 

0.006148 

4.28 

<.0001* 

|CIWS/PDMS(1]*(RedAircrafts-3.50195) 

-0.026277 

0.008031 

3 ll 

- , 

|CIWS/PDMS{2)*(RedAircrafts-3. 50195) 

0.0262774 

0.008031  3.27  0.0012*  \ 

tlW^PdM5tljJ(yiue^ipLong5SK^g^TMS3J 

6.748 5e-T  it!  67e-7 

437 

<70or 

CIWS/PDMS[21*(BlueShipLongSAMRng-1 05000) 

-6.249e-7 

1.367e-7 

-4.57 

<.0001* 

IUAV[0]  *  ( RedA  i  rcrafts-3 . 50 1 95) 

-0.005478 

0.007992 

-0.69 

0.4938  1 

!UAV[1  ]*(RedAircrafts-3. 50195) 

0.0054776 

0.007992 

0.69 

0.4938  ! 

(BlueShipRadarRng-145000)*(RedAircrafts-3.50195) 

-4.144e-7 

1.419e-7 

-2.92 

0.0038* 

(LandBasedASMs-4.00778)*(RedAircrafts-3.50195) 

0.0180938 

0.006959 

2.60 

0.0099* 

(LandBasedASMs-4.00778)*(BlueShipMedSAMRng-47500) 

4.4475e-7 

3.001  e-7 

1.48 

0.1397 

(LandBasedASMs-4.00778)*(BlueShipLongSAMRng-105000) 

-2.422e-7 

1.024e-7 

-2.37 

0.0188* 

(BlueShipShortSAM  Rng-20000)*(BlueShipMedSAMRng-47500) 

-1.67e-11 

4.06e-11 

-0.41 

0.6804 

(BlueShipMedSAMRng-47500)*(BlueShipLongSAMRng-105000) 

-1 .1 8e-1 1 

6.7e-12 

-1.76 

0.0804 

(BlueShipRadarRng-145000)*(BlueShipRadarRng- 145000) 

-9.14e-12 

2.44  e-12 

-3.75 

0.0002* 

SAM  Option^l^ 

-0.091086 

0.0119 

-7.65 

<-000 V 

jSAM  Ogtion|2^ 

0.0248864 

0.010801 

2.30 

i*l 

SAM  Option[3] 

0.0661996 

0.011839 

5.59 

<  0001'  ) 

Figure  25.  Expanded  Estimates  for  the  Second  Order  Model. 


c.  Main  Effects  Model  for  the  Number  of  Enemy  Casualties 

Using  the  stepwise  regression  technique  with  AlCc  criterion,  the  factors 
shown  in  Figure  26  are  significant  enough  to  be  a  predictor  of  the  number  of 
enemy  causalities.  Differing  from  the  main  effects  model  for  frigate  survivability, 
the  range  of  the  short  range  SAM  and  number  of  land  attack  missiles  are  not 
significant  predictors  in  this  model. 
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Source 

LogWorth 

CIWS/PDMS 

63.444 

BlueShipLongSAMRng 

28.040 

RedAircrafts 

17.037 

SAM  Option 

16.221 

BlueShipMedSAMRng 

4.315 

BlueShipRadarRng 

3.543 

PValue 

0.00000 

0.00000 

0.00000 

0.00000 

0.00005 

0.00029 


Figure  26.  Effect  Summary  of  the  Factors  for  Main  Effect  Model. 


The  actual  by  predicted  plot  and  the  summary  of  fit  of  the  main  effects 
model  is  as  shown  in  Figure  27.  The  R  squared  value  is  around  0.787  for  this 
model. 


Actual  by  Predicted  Plot 


j 

< 

•X 


<T5 

-r> 

ro 

U 

>s 

E 

ii 

c 

UJ 

c 

1 

<b 

2 


Mean(Enemy  Casualties)  Predicted  P<.0001 
RSq=0.79  RMSE=1.7771 


Summary  of  Fit 

RSquare 

0.786789 

RSquare  Adj 

0.780795 

Root  Mean  Square  Error 

1.777106 

Mean  of  Response 

8.748444 

Observations  (or  Sum  Wgts) 

257 

Figure  27.  Actual  by  Predicted  Plot  and  the  Summary  of  Fit  for  the  Main 

Effects  Model  for  Enemy  Casualties. 


Expanded  parameter  estimates  appear  in  Figure  28.  According  to  t  ratio 
values,  CIWS/PDMS  selection,  range  of  long  range  SAM,  SAM  option  ,and 
number  of  aircraft  have  more  significant  effects  as  compared  to  radar  range  and 
medium  range  SAM’s  range. 
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Expanded  Estimates 

Nominal  factors  expanded  to  all  levels 

Term 

Estimate 

Std  Error 

t  Ratio 

Prob>|t| 

Intercept 

0.1707908 

0.58806 

0.29 

0.7717 

SAM  Option[1] 

-1.616312 

0.173729 

-9.30 

<•0001* 

SAM  Option [2] 

0.4928158 

0.147833 

3.33 

0.0010* 

SAM  Option[3] 

1.1234958 

0.172975 

6.50 

<.0001* 

CIWS/PDMS[1] 

-2.577384 

0.111108 

-2320 

<.0001* 

CIWS/PDMS[2] 

2.5773843 

0.111108 

2320 

<.0001* 

BlueShipRadarRng 

6.7183e-6 

1 ,826e-6 

3.68 

0.0003* 

Red  A  ire  rafts 

1.0766072 

0.116137 

9.27 

<.0001* 

BlueShipMedSAMRng 

2.1 133e-5 

5.11e-6 

4.14 

<.0001* 

BlueShiplongSAMRng 

2.5641e-5 

2.022e-6 

12.68 

<.0001* 

Figure  28.  Expanded  Estimates  for  the  Main  Effects  Model. 


d.  Second  Order  Model  for  the  Number  of  Enemy  Casualties 

To  evaluate  the  effects  of  two-way  interactions  and  polynomial  terms,  we 
build  the  second  order  regression  model  for  the  number  of  enemy  casualties 
using  the  same  regression  technique  and  AlCc  criterion.  Predictors  used  in  this 
model  appear  in  Figure  29. 
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Source 

CIWS/PDMS 
BlueShipLongSAMRng 
Red  A  ire  rafts 
SAM  Option 

CIWS/PDMS’RedAircrafts 

CIWS/PDMS’LandBasedASMs 

BlueShipMedSAMRng 

BlueShipRadarRng 

BlueShipRadarRng’BlueShipRadarRng 

CIWS/PDMS*BlueShipLongSAMRng 

RedAircrafts’BlueShipLongSAMRng 

Red  A  i  rc  rafts*  Red  A  i  rc  rafts 

BlueShipRadarRng’BlueShipLongSAMRng 

BlueShipLongSAMRng’BlueShipLongSAMRng 

CIWS/PDMS'UAV 

UAV 

CIWS/PDMS’BlueShipRadarRng 

BlueShipMedSAMRng*BlueShipMedSAMRng 

UAV*BlueShipRadarRng 

BlueShipRadarRng*LandBasedASMs 

Red  A  i  rc  rafts*  B I  u  eSh  i  p  M  ed  SA  M  Rng 

BlueShipShortSAMRng 

LandBasedASMs 

BlueShipShortSAMRng*BlueShipMedSAMRng 

BlueShipRadarRng’RedAircrafts 


LogWorth 


PValue 

0.00000 

0.00000 

0.00000 

0.00000 

0.00000 

0.00000 

0.00000 

0.00001 

0.00002 

0.00010 

0.00162 

0.01389 

0.01629 

0.03170 

0.05109 

0.08470  A 

0.08935 

0.10487 

0.18409 

0.19097 

0.19767 

0.25707 

0.40019  A 

0.69680 

0.95243 


Figure  29.  Effect  Summary  of  the  Factors  for  Second  Order  Model. 


The  R  squared  value  is  around  0.88  for  the  second  order  model.  Figure  30 
displays  the  summary  of  fit  and  actual  by  predict  plot.  As  shown,  CIWS/PDMS 
selection,  range  of  long  range  SAM,  number  of  enemy  aircraft,  SAM  option  and 
CIWS/PDMS  x  number  of  enemy  aircraft  are  the  five  most  significant  factors  in 
predicting  enemy  casualties. 
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Actual  by  Predicted  Plot 


0  2  4  6  8  10  12  14  16  18 

Mean(Enemy  Casualties)  Predicted  P<.0001 


RSq=0.88  RMSE=1.378 


Summary  of  Fit 

RSquare 

0.881582 

RSquare  Adj 

0.868195 

Root  Mean  Square  Error 

1.378015 

Mean  of  Response 

8.748444 

Observations  (or  Sum  Wgts) 

257 

Figure  30.  Actual  by  Predicted  Plot  and  the  Summary  of  Fit  for  the 

Second  Order  Model. 


5.  Classification  and  Regression  Tree 

Classification  and  regression  trees  offer  an  easy  way  to  examine  the 
contribution  of  the  factors  to  the  outcome.  Classification  trees  are  used  when  the 
response  or  outcome  is  discrete  or  categorical.  If  the  response  is  a  continuous 
variable,  regression  trees  are  used.  We  use  regression  trees  for  analysis  of  the 
factors  that  affect  the  frigate’s  survival  probability  using  the  summarized  data. 

After  building  our  regression  tree  model,  we  examine  the  candidate  split 
point  reports  generated  by  JMP  (Figure  31),  which  shows  the  LogWorth  value  of 
the  factors.  The  split  occurs  according  to  the  LogWorth  statistic.  For  example,  the 
CIWS/PDMS  factor  is  the  first  optimal  split  point  in  the  model. 
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Candidates 

Term 

Candidate  SS 

LogWorth  Cut  Point 

SAM  Option 

1.10712788 

2.1060455  1 

riWS/PDMS 

22.77077263  * 

1 1 3*4~92899  1  ! 

UAV 

0.00655945 

0.0712817  0 

BlueShipRadarRng 

1 .77272841 

2.6724219  82656 

LandBasedASMs 

1.28775413 

2.3612961  5 

Red  A  ire  rafts 

1.41975858 

2.7843386  4 

BlueShipShortSAMRng 

0.54705871 

0.4484654  21875 

BlueShipMedSAMRng 

0.77409554 

0.7894175  33438 

BlueShipLongSAMRng 

1 .84783974 

2.8303513  79023 

Figure  31 .  Candidate  Split  Points. 


The  regression  tree  appears  in  Figure  32.  As  stated  earlier,  the  first  split 
occurs  with  the  CIWS/PDMS  selection  factor.  If  CIWS  is  selected  in  the  design 
the  survival  probability  becomes  0.25;  however,  it  increases  up  to  0.85  if  PDMS 
is  employed. 


Figure  32.  Regression  Tree  for  Frigate’s  Survival  Probability. 


The  second  partition  occurs  under  selection  of  CIWS  (CIWS/PDMS=1) 

with  the  range  of  long  range  SAM.  This  means  that  if  CIWS  is  employed  and  the 
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range  of  the  long  range  SAM  is  less  than  153,984  meters,  the  probability  of 
survival  becomes  0.18.  If  the  range  is  above  153,984,  the  survival  probability  is 
0.46.  It  is  also  shown  on  the  right  leaf  of  the  tree  that  if  PDMS  is  employed  and 
the  number  of  aircraft  is  less  than  or  equal  to  four,  the  frigate’s  survival  rate 
becomes  0.90. 

After  seven  splits  we  reach  an  R  square  value  of  0.842.  An  increase  in  the 
number  of  splits  always  yields  an  increase  in  the  R  squared  value.  Furthermore, 
more  splits  cause  additional  complexity  in  the  tree,  and  it  does  not  contribute  to 
the  model  significantly.  Figure  33  displays  the  split  history  (number  of  splits  vs.  R 
squared  value).  After  the  seventh  split,  more  splitting  does  not  contribute  to  our 
model  in  terms  of  R  squared  value. 


0  2  4  6  8  10  12 

Figure  33.  Split  Histo ry. 

To  evaluate  the  contributions  of  the  factors,  we  need  to  examine  the 
column  contributions  report  created  by  JMP  (Figure  34).  As  it  is  easily  seen  in 
this  report,  the  most  significant  five  factors  are  the  same  as  the  previous  ones. 
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Column  Contributions 

Number 


Term 

CIWS/PDMS 

BlueShipLongSAMRng 

LandBasedASMs 

Red  A  ire  rafts 

SAM  Option 

UAV 


of  Splits 

1 

1 

2 

2 

1 

0 


ss 

22.7707726  HZ 
2.01047458  □ 
1.8149372  □ 

1 .24843863  ] 
0.77026955  ] 

0 


Portion 

0.7958 

0.0703 

0.0634 

0.0436 

0.0269 

0.0000 


Figure  34.  Contributions  of  the  Factors. 


D.  FACTOR  SIGNIFICANCE 

The  significance  of  specific  design  factors  in  regard  to  a  frigate’s 
survivability  and  the  number  of  enemy  casualties  is  summarized  in  the  following 
subsections. 

1.  Factor  Significance  in  Frigate’s  Survival  Probability 

For  the  particular  AAW  scenario  in  this  thesis,  factors  that  are  determined 
to  be  significant  in  a  frigate’s  survivability  and  their  rankings  are  as  shown  in 
Table  1 1 . 
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Table  1 1 .  Summary  of  the  Factor  Significance  for  a  Frigate’s  Survival 


Probability. 

Factor  Name 

Main  Effects 

Second 

Regression 

Model 

Order  Model 

Tree 

CIWS/PDMS  Selection 

1 

1 

1 

Range  of  Long  Range  SAM 

2 

2 

2 

Number  of  Enemy  Aircraft 

3 

3 

4 

Number  of  Land  Based  ASMs 

4 

4 

3 

SAM  Option 

5 

5 

5 

Range  of  Medium  Range  SAM 

6 

10 

- 

Radar  Range 

7 

8 

- 

UAV 

_ 

18 

_ 

The  selection  of  the  CIWS/PDMS  is  the  most  significant  design  factor. 
Employment  of  the  PDMS  is  superior  to  CIWS  in  terms  of  a  frigate’s  survivability. 
This  makes  sense  as  PDMS  has  higher  probability  of  hit  compared  to  CIWS. 

The  range  of  the  long  range  SAM  is  the  second  most  significant  design 
factor  that  affects  the  frigate’s  survivability.  An  increase  in  the  range,  increases 
survivability. 

The  third  and  fourth  most  important  factors  are  the  number  of  enemy 
aircraft  and  land-based  ASMs.  It  is  obvious  that  if  the  number  of  enemy  assets 
increases,  the  survival  probability  of  the  friendly  frigate  decreases.  These  are 
uncontrollable  factors,  but  they  can  be  estimated  according  to  the  planned 
operational  employment  of  the  prospective  frigates. 

SAM  option  is  the  other  factor  that  has  a  significant  effect  on  the  friendly 

frigate’s  survivability.  Although  SAM  option  1  decreases  the  survivability, 

selection  of  SAM  option  2  or  SAM  option  3  increases  the  survival  probability. 
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Among  all  SAM  options,  the  selection  of  long  range  and  medium  range  SAM 
(option  3)  has  the  most  positive  effect  on  survivability  of  the  frigate.  This  makes 
sense,  as  it  is  better  to  counter  threats  at  longer  ranges.  Range  of  the  radar  and 
the  medium  range  SAM  are  additional  significant  factors. 

Use  of  the  UAV  in  AAW  operation  does  not  have  a  significant  effect  on  the 
frigate’s  survivability.  Our  purpose  of  including  a  UAV  in  the  scenario  is  to 
explore  whether  a  UAV  provides  early  warning  for  the  frigate  to  counteract  an 
enemy  threat.  As  a  result  of  the  analysis,  the  presence  of  the  UAV  does  not 
appear  to  be  a  significant  factor  in  AAW  mission  in  terms  of  frigate’s  survivability. 
Longer  range  UAVs  with  greater  surveillance  capabilities  may  have  more  effect, 
but  they  are  not  explored  in  this  study. 

2.  Factor  Significance  in  the  Number  of  Enemy  Casualties 

Table  12  displays  the  significant  factors  in  the  number  of  enemy  casualties 
and  their  rankings.  The  results  are  almost  the  same  as  the  previous  ones. 

Table  1 2.  Summary  of  the  Factor  Significance  for  Number  of  Enemy 

Casualties. 


Factor  Name 

Main  Effects  Model 

Second  Order  Model 

CIWS/PDMS  Selection 

1 

1 

Range  of  Long  Range  SAM 

2 

2 

Number  of  Enemy  Aircraft 

3 

3 

Number  of  Land  Based  ASMs 

4 

- 

SAM  Option 

5 

4 

Range  of  Medium  Range  SAM 

6 

7 

Radar  Range 

7 

8 

UAV 

- 

16 
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V.  CONCLUSION 


A.  SUMMARY 

This  research  investigates  the  effectiveness  of  combinations  of  weapons 
and  sensors  onboard  a  frigate  in  an  AAW  environment.  It  also  evaluates  the 
effectiveness  of  an  unmanned  aerial  vehicle  for  an  AAW  mission.  By  doing  so 
our  aim  is  to  determine  the  needed  weapon  and  sensor  capabilities  before  ship 
hull  design  is  complete  to  make  operational  effectiveness  independent  from 
physical  design  considerations. 

An  AAW  scenario  was  built  to  evaluate  the  candidate  weapons,  sensor 
capabilities,  and  use  of  a  prospective  UAV  in  an  AAW  operation.  We  ran  25,700 
simulated  AAW  battles  in  MANA,  and  the  resulting  data  were  imported  to  the 
JMP  statistical  discovery  program  for  analysis  purposes.  Due  to  limitations  of 
MANA,  a  few  characteristics  of  the  AAW  environment,  such  as  the  flight  pattern 
of  the  missiles  and  the  altitude  of  the  aircraft,  could  not  be  represented.  We  also 
made  assumptions  related  to  enemy  tactics  and  the  operational  environment. 
Therefore,  this  study  could  not  answer  all  questions  related  to  the  capabilities 
needed  for  AAW  operations.  It  does,  however,  provide  useful  insights  about 
weapon  and  sensor  employment  onboard  an  AAW  frigate. 

The  result  of  this  analysis  shows  that  the  most  important  design  factor  for 
frigate  AAW  operations  is  the  selection  of  CIWS  or  PDMS.  Moreover,  it  shows 
that  PDMS  is  superior  to  CIWS.  This  study  also  posits  that  the  range  of  the 
SAMs,  the  combination  of  the  SAMs  onboard,  and  radar  range  have  significant 
impacts  on  the  success  of  an  AAW  operation.  In  addition,  this  research  provides 
evidence  that  the  use  of  a  medium  range  UAV  in  an  AAW  environment  does  not 
significantly  contribute  to  mission  success. 

B.  ANSWERING  RESEARCH  QUESTIONS 

The  research  questions  introduced  in  the  beginning  of  the  thesis  are  as 
follows: 
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1 .  Among  a  set  of  air  defense  weapon  systems  alternatives, 
what  is  the  most  effective  combination  for  a  future  AAW 
frigate  design? 

2.  How  effective  are  Point  Defense  Missile  Systems  (PDMS) 
compared  to  Close-in  Weapon  Systems  (CIWS)  with 
different  weapon  configurations? 

3.  Does  employing  a  prospective  unmanned  aerial  vehicle 
(UAV)  onboard  AAW  frigates  have  significant  advantages  in 
an  AAW  mission? 

4.  What  is  the  probability  of  survivability  against  enemy  air 
assets  with  different  weapon  combinations? 

5.  What  are  the  strengths  and  drawbacks  of  utilizing  Map 
Aware  Non-Uniform  Automata  (MANA)  to  construct  realistic 
scenarios  for  evaluating  AAW  effectiveness  of  naval  ships? 

To  answer  the  first  question,  we  vary  the  SAM  options  and  presence  of 
CIWS  or  PDMS  onboard  a  frigate.  As  a  result,  long  range  and  medium  range 
SAMs  and  the  PDMS  are  the  best  mix  of  air  defense  weapon  systems,  in 
addition  to  the  main  gun.  The  range  of  the  long  range  missile  should  be  more 
than  154,000  meters. 

For  the  second  question,  we  vary  the  presence  of  PDMS  and  CIWS 
onboard  a  frigate.  The  PDMS’  defensive  capabilities  are  superior  to  CIWS  in 
terms  of  both  survivability  and  number  of  enemy  casualties.  It  is  also  discovered 
in  the  analysis  that  selection  of  the  CIWS  has  the  greatest  negative  impact  on  the 
survival  probability. 

Interestingly,  employment  of  a  UAV  for  an  AAW  mission  does  not 
contribute  to  success  of  the  operation.  In  each  analysis,  presence  of  the  UAV 
was  either  excluded  from  the  significant  factors  or  it  has  less  significance  on  the 
response  surface. 

The  previously  stated  mix  of  the  weapons  provides  the  best  survival  rate. 
Selection  of  the  short  range  SAM  rather  than  medium  range  SAM  reduces  the 
probability  of  survival,  but  it  still  provides  a  fairly  good  survival  rate.  By  contrast, 
employment  of  CIWS  instead  of  PDMS  or  selecting  the  pair  of  medium  and  short 

50 


SAMs  instead  of  long  and  medium  range  SAMs  causes  an  extreme  decrease  in 
the  survival  probability. 

Addressing  the  final  question,  MANA  has  many  strengths  in  combat 
simulation.  It  has  a  straightforward  user  interface  to  simulate  many  aspects  of 
combat,  such  as  setting  up  a  weapon  and  sensor  capabilities,  as  well  as  defining 
the  communication  lines’  characteristics.  Nonetheless,  it  has  several  drawbacks, 
especially  those  related  to  aerial  platforms  and  weapons.  For  example, 
simulation  of  the  flight  pattern  of  the  missiles  and  altitude  of  the  aircraft  are  not 
included  in  MANA.  The  effectiveness  settings  of  a  gun  with  high  rate  of  fire  could 
also  be  more  straightforward  in  the  model  setup. 

C.  FURTHER  RESEARCH 

Classified  information  is  not  included  in  this  study.  For  future  work, 
classified  information  such  as  probability  of  hit  for  a  particular  weapon  or  the 
detection  probability  of  radars  can  be  included  in  the  model  to  get  more  precise 
results. 

We  explore  the  effectiveness  of  a  prospective  UAV  in  an  AAW 
environment  for  detection  purposes.  These  UAVs  can  be  loaded  with  weapons 
and  they  can  serve  in  any  environment.  For  future  models,  armed  UAVs  can  be 
simulated  in  a  multi-threat  (air,  surface,  and  sub-surface  threats)  environment. 
And,  as  previously  mentioned,  longer  range  UAVs  with  greater  sensor 
capabilities  can  be  assessed  for  their  value  to  an  AAW  mission. 
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